The present invention relates to a system and method for generating a wide-area high-resolution image by capturing partial images (sub-images) of parts of a target object such as a document and thereafter connecting or synthesizing the sub-images.
Description of the Related Art
When a document of a large size is captured and inputted as image data by use of a capture device (video camera etc.) in a high resolution that is suitable for character recognition, the document is generally segmented into parts and the segmented parts are captured individually by the capture device to be synthesized or connected later. An example of such a conventional wide-area high-resolution image generation method has been disclosed in Japanese Patent Application Laid-Open No. HEI10-210272 (hereafter, referred to as xe2x80x9cdocument No. 1xe2x80x9d). In the conventional technique of the document No. 1, a wide-area high-resolution image is obtained as described below.
First, the whole of a document (the target object of capturing) is preliminarily captured by a camera and thereby an image of the whole of the document (hereafter, referred to as a xe2x80x9ctotal imagexe2x80x9d) is obtained. Subsequently, the number of sub-images (images of parts of the document) necessary for covering the whole of the document with a predetermined resolution is determined and thereby a zoom ratio (power) to be used for capturing the sub-images is determined. Subsequently, each part of the document (including the periphery (margin) of the part) is successively captured by the camera with the zoom ratio determined above, and thereby sub-images of the parts of the document are obtained. Subsequently, adjoining sub-images are successively connected together by use of image information (pixel intensity, structure such as lines, etc.) and thereby a wide-area high-resolution image of the whole of the document is generated.
A method which is similar to the wide-area high-resolution image generation method of the document No. 1 has been disclosed in a document: Noboru Nakajima, Naoya Tanaka and Keiji Yamada xe2x80x9cDocument Reconstruction and Recognition from an Image Sequence,xe2x80x9d Proceedings 14th International Conference on Pattern Recognition (IAPR (International Association for Pattern Recognition)), Vol. 1 pp. 922-925 (1998) (hereafter, referred to as xe2x80x9cdocument No. 2). In the method of the document No. 2, the connection of the adjoining high-resolution sub-images is conducted at small calculation cost, by executing structure analysis to each sub-image and using structure information (hierarchical layout structure such as, characters, character strings, text blocks (paragraphs), columns, etc.) of the sub-images.
However, the conventional wide-area high-resolution image generation methods which have been described above involves the following problems or drawbacks. First, if a document having some large blank areas is segmented into parts and sub-images are obtained by capturing parts of the document, some of the sub-images might be totally blank or might be images having blank margins. In such cases, the connection of the adjoining sub-images and the generation of the wide-area high-resolution image become impossible since the connecting margins of such sub-images do not have structure information (lines, characters, etc.) to be used for the registration. Therefore, the conventional methods are not capable of generating a wide-area high-resolution image when a relatively large blank area exists in the document or in the originally captured total image.
Further, the two adjoining sub-images to be connected together are generally required to have the same resolution. Therefore, in the conventional methods (which successively connect adjoining sub-images for obtaining the wide-area high-resolution image), all the sub-images have to be captured in the same resolution. If a document to be captured includes a short text or character string that has to be captured in a high resolution for character recognition etc., the whole of the document has to be segmented into a lot of small parts according to the required high resolution and all the parts have to be captured in the same high resolution, taking an enormous processing time.
It is therefore the primary object of the present invention to provide a system and method for generating a wide-area high-resolution image, by which the wide-area high-resolution image can successfully be generated even if a relatively large blank area existed in the document to be captured or in the originally captured total image.
Another object of the present invention is to provide a system and method for generating a wide-area high-resolution image, by which the wide-area high-resolution image can be generated by use of images of different resolutions.
Another object of the present invention is to provide a system and method for generating a wide-area high-resolution image, which can decrease the number of necessary sub-images and shorten the processing time when one or more areas (such as blank areas) that do not require high resolution capturing existed in the originally captured total image.
In accordance with a first aspect of the present invention, there is provided a wide-area high-resolution image generation method comprising a total image acquisition step, an image structure analysis step, a sub-image acquisition step, an image connection step, an image extraction step and a wide-area high-resolution image generation step. In the total image acquisition step, a target object of the generation of a wide-area high-resolution image is captured by a capture device and thereby a total image of the target object is acquired. In the image structure analysis step, image structure analysis is conducted to the total image of the target object and thereby structural elements are extracted from the total image and position information of each structural element is obtained. The sub-image acquisition step is conducted for one or more of the structural elements. In the sub-image acquisition step, one or more partial areas and a resolution to be used for capturing the structural element are determined and sub-images of the partial areas of the structural element are acquired by the capture device with the determined resolution. The image connection step is conducted for each of the structural elements to which the sub-image acquisition step has been conducted. In the image connection step, the sub-images of the partial areas of the structural element are connected together by use of image information of the sub-images and thereby an image of the structural element having the determined resolution is obtained as a synthesis target image. The image extraction step is conducted for each of the structural elements to which the sub-image acquisition step has not been conducted. In the image extraction step, part of the total image corresponding to the structural element is extracted from the total image as a synthesis target image. In the wide-area high-resolution image generation step, the synthesis target images of the structural elements obtained in the image connection steps and the image extraction steps are synthesized so that relative position relationship of the synthesis target images will be the same as that of the structural elements in the total image of the target object based on the position information of the structural elements obtained in the image structure analysis step and thereby a wide-area high-resolution image of the target object is obtained.
In accordance with a second aspect of the present invention, in the first aspect, the wide-area high-resolution image generation method further comprises an attribute determination step. In the attribute determination step, the attribute of each structural element is determined based on the image structure analysis. The sub-image acquisition step is conducted for structural elements having attributes that require higher resolution than that of the total image.
In accordance with a third aspect of the present invention, in the second aspect, the wide-area high-resolution image generation method further comprises a resolution judgment step. The resolution judgment step is conducted for each of the structural elements to which the sub-image acquisition step has been conducted. In the resolution judgment step, whether or not a sufficient resolution predetermined for the attribute of the structural element could be attained is judged. For each of the structural elements that have been judged to have insufficient resolution in the resolution judgment step, the sub-image acquisition step is repeated with a higher resolution and new partial areas until the sufficient resolution predetermined for the attribute is attained. The image connection step for the structural element is conducted by use of the sub-images which attained the sufficient resolution.
In accordance with a fourth aspect of the present invention, in the resolution judgment step in the third aspect, one or more fine structural elements are extracted from the sub-images of the structural element and the judgment on the resolution is executed based on pixel density of the extracted fine structural elements.
In accordance with a fifth aspect of the present invention, in the fourth aspect, one or more letters are extracted as the fine structural elements in the resolution judgment step.
In accordance with a sixth aspect of the present invention, in the first aspect, the wide-area high-resolution image generation method further comprises a geometrical deformation estimation step and a geometrical deformation compensation step. In the geometrical deformation estimation step, geometrical deformation is estimated between every two adjoining sub-images of a structural element and thereby geometrical deformation of each sub-image of the structural element is estimated. In the geometrical deformation compensation step, the geometrical deformation of each sub-image of the structural element is compensated for based on the geometrical deformation estimated in the geometrical deformation estimation step and thereby deformation-compensated sub-images of the structural element are obtained to be used in the image connection step.
In accordance with a seventh aspect of the present invention, in the geometrical deformation estimation step in the sixth aspect, the estimation of the geometrical deformation between two adjoining sub-images is conducted using one or more fine structural elements extracted from the two adjoining sub-images.
In accordance with an eighth aspect of the present invention, in the seventh aspect, one or more letters extracted from the two adjoining sub-images are used as the fine structural elements in the geometrical deformation estimation step.
In accordance with a ninth aspect of the present invention, in the seventh aspect, the estimation of the geometrical deformation between the two adjoining sub-images in the geometrical deformation estimation step is conducted by estimating geometrical deformation of each of one or more fine structural elements between the two adjoining sub-images individually and taking the average of the geometrical deformations of the fine structural elements.
In accordance with a tenth aspect of the present invention, in the fourth aspect, the wide-area high-resolution image generation method further comprises a geometrical deformation estimation step and a geometrical deformation compensation step. In the geometrical deformation estimation step, geometrical deformation is estimated between every two adjoining sub-images of a structural element and thereby geometrical deformation of each sub-image of the structural element is estimated. In the geometrical deformation compensation step, the geometrical deformation of each sub-image of the structural element is compensated for based on the geometrical deformation estimated in the geometrical deformation estimation step and thereby deformation-compensated sub-images of the structural element are obtained to be used in the image connection step. In the geometrical deformation estimation step, the estimation of the geometrical deformation between two adjoining sub-images is conducted using the fine structural elements extracted in the resolution judgment step.
In accordance with an eleventh aspect of the present invention, there is provided a wide-area high-resolution image generation system comprising a processing device which is connected to a capture device whose capturing direction and zoom ratio are controllable. The processing device includes a total image acquisition means, an image structure analysis means, a sub-image acquisition means, an image connection means, an image extraction means and a wide-area high-resolution image generation means. The total image acquisition means captures a target object of the generation of a wide-area high-resolution image by use of the capture device and thereby acquires a total image of the target object. The image structure analysis means conducts image structure analysis to the total image of the target object, and thereby extracts structural elements from the total image and obtains position information of each structural element. The sub-image acquisition means conducts a sub-image acquisition process for one or more of the structural elements. In the sub-image acquisition process, one or more partial areas and a resolution to be used for capturing the structural element are determined and sub-images of the partial areas of the structural element are acquired by the capture device with the determined resolution. The image connection means conducts an image connection process for each of the structural elements to which the sub-image acquisition process has been conducted. In the image connection process, the sub-images of the partial areas of the structural element are connected together by use of image information of the sub-images and thereby an image of the structural element having the determined resolution is obtained as a synthesis target image. The image extraction means conducts an image extraction process for each of the structural elements to which the sub-image acquisition process has not been conducted. In the image extraction process, part of the total image corresponding to the structural element is extracted from the total image as a synthesis target image. The wide-area high-resolution image generation means synthesizes the synthesis target images of the structural elements obtained in the image connection process and the image extraction process so that relative position relationship of the synthesis target images will be the same as that of the structural elements in the total image of the target object based on the position information of the structural elements obtained by the image structure analysis means, and thereby obtains a wide-area high-resolution image of the target object.
In accordance with a twelfth aspect of the present invention, in the eleventh aspect, the processing device further includes an attribute determination means for determining the attribute of each structural element based on the image structure analysis conducted by the image structure analysis means. The sub-image acquisition means conducts the sub-image acquisition process for structural elements having attributes that require higher resolution than that of the total image.
In accordance with a thirteenth aspect of the present invention, in the twelfth aspect, the processing device further includes a resolution judgment means which conducts a resolution judgment process for each of the structural elements to which the sub-image acquisition process has been conducted. In the resolution judgment process, whether or not a sufficient resolution predetermined for the attribute of the structural element could be attained is judged. For each of the structural elements that have been judged to have insufficient resolution in the resolution judgment process, the sub-image acquisition means repeats the sub-image acquisition process with a higher resolution and new partial areas until the sufficient resolution predetermined for the attribute is attained. The image connection means conducts the image connection process for the structural element by use of the sub-images which attained the sufficient resolution.
In accordance with a fourteenth aspect of the present invention, in the thirteenth aspect, the resolution judgment means extracts one or more fine structural elements from the sub-images of the structural element and executes the judgment on the resolution based on pixel density of the extracted fine structural elements.
In accordance with a fifteenth aspect of the present invention, in the fourteenth aspect, the resolution judgment means extracts one or more letters as the fine structural elements.
In accordance with a sixteenth aspect of the present invention, in the eleventh aspect, the processing device further includes a geometrical deformation estimation means and a geometrical deformation compensation means. The geometrical deformation estimation means estimates geometrical deformation between every two adjoining sub-images of a structural element and thereby estimates geometrical deformation of each sub-image of the structural element. The geometrical deformation compensation means compensates for the geometrical deformation of each sub-image of the structural element based on the geometrical deformation estimated by the geometrical deformation estimation means, and thereby obtains deformation-compensated sub-images of the structural element to be used in the image connection process.
In accordance with a seventeenth aspect of the present invention, in the sixteenth aspect, the geometrical deformation estimation means conducts the estimation of the geometrical deformation between two adjoining sub-images using one or more fine structural elements extracted from the two adjoining sub-images.
In accordance with an eighteenth aspect of the present invention, in the seventeenth aspect, the geometrical deformation estimation means uses one or more letters extracted from the two adjoining sub-images as the fine structural elements.
In accordance with a nineteenth aspect of the present invention, in the seventeenth aspect, the geometrical deformation estimation means conducts the estimation of the geometrical deformation between the two adjoining sub-images by estimating geometrical deformation of each of one or more fine structural elements between the two adjoining sub-images individually and taking the average of the geometrical deformations of the fine structural elements.
In accordance with a twentieth aspect of the present invention, in the fourteenth aspect, the processing device further includes a geometrical deformation estimation means and a geometrical deformation compensation means. The geometrical deformation estimation means estimates geometrical deformation between every two adjoining sub-images of a structural element and thereby estimates geometrical deformation of each sub-image of the structural element. The geometrical deformation compensation means compensates for the geometrical deformation of each sub-image of the structural element based on the geometrical deformation estimated by the geometrical deformation estimation means, and thereby obtains deformation-compensated sub-images of the structural element to be used in the image connection process. The geometrical deformation estimation means conducts the estimation of the geometrical deformation between two adjoining sub-images using the fine structural elements extracted by the resolution judgment means.
In accordance with twenty-first through thirtieth aspects of the present invention, there are provided machine-readable record mediums storing programs for instructing a computer, a DSP (Digital Signal Processor), etc. to execute the wide-area high-resolution image generation methods of the first through tenth aspects of the present invention.